Skip to content

feat(gateway): integrate the Wave 6 gateway features - #69

Merged
ALikesToCode merged 54 commits into
integrate/gateway-waves-1-5from
integrate/gateway-waves-1-6
Oct 9, 2026
Merged

ALikesToCode merged 54 commits into
integrate/gateway-waves-1-5from
integrate/gateway-waves-1-6

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

This branch merges all eight Wave 6 feature branches on top of the Wave 5 integration branch (#60) and wires them together. It is based on integrate/gateway-waves-1-5, so this diff shows only Wave 6.

PR Feature Setting
#61 Latency SLO admission LATENCY_SLO_MODE, LATENCY_SLO_POLICY_JSON
#65 Bounded hedged auto requests HEDGED_REQUESTS_ENABLED, HEDGED_REQUESTS_POLICY_JSON
#62 Worker Realtime WebSocket transport REALTIME_ENABLED, REALTIME_PROVIDERS_JSON, migration 0029
#66 Per-key stream cost breaker STREAM_COST_BREAKER_ENABLED, migration 0030
#67 Learned provider cooldowns LEARNED_COOLDOWN_MODE, LEARNED_COOLDOWN_POLICY_JSON, migration 0031
#68 Signed usage receipts USAGE_RECEIPTS_ENABLED, USAGE_RECEIPTS_KEY_ID, USAGE_RECEIPTS_SIGNING_KEY_REF, migration 0032
#64 Context-leak canary tokens CONTEXT_CANARY_MODE, CONTEXT_CANARY_POLICY_JSON
#63 Enterprise integration contracts ENTERPRISE_PREVIEW_ENABLED

The four migration PRs form one stack (#62 → #66 → #67 → #68) so each keeps its numbering contiguous. All eight are merged here.

Integration changes

Flask hook order

The named post-authentication order gains two hooks:

  1. Request policy: configuration security freshness and retention (existing).
  2. Prompt injection (existing).
  3. Batch spillover (existing).
  4. PII redaction (existing).
  5. Hosted Responses state (existing).
  6. Context canary scope. Records whether the authenticated key and route opted in. It never changes the caller's body, and a later pre-dispatch rejection clears the scope.
  7. Generation deadline (existing).
  8. Latency SLO admission. A rejection returns 503 before any idempotency claim, admission lease or provider call.
  9. Idempotency claim or replay (existing).
  10. Admission (existing).
  11. Dispatch through the managed pipeline.

app.py also mounts the usage receipt routes. While receipts are off, a gate that runs before authentication returns a JSON 404 with Cache-Control: no-store, so an unauthenticated caller can't tell whether the routes exist.

Storage

  • The Flask D1 store reaches two new private endpoints: learned-cooldown (4 KiB) and usage-receipts (128 KiB).
  • After a reservation reconciliation is applied, both runtimes write one receipt per settled event, keyed by the immutable transition ID. A receipt failure never changes the reconciliation response.

Managed dispatch

  • Latency candidates: the new services/latency_slo_candidates.py returns the reviewed eligible candidates for an auto route. It reads storage only: it never seeds routes, claims a session-lane lease or chooses a health probe or cohort. Whatever it can't resolve has unknown coverage, as the doc says.
  • Latency reroute: later selection keeps out every candidate the early decision removed, and hedging races only the approved candidates.
  • Context canary:
    • The annotation is added to the serialized provider bytes after the submission guard. The idempotency fingerprint, caches, hosted state and logs never see it.
    • Opted-in requests bypass the exact, shared and semantic caches.
    • The scan runs in _finish_turn before any other finalizer, and inside each hedged attempt with its own context.
    • The intelligence transport's detached send thread gets the same scan through a wrapper.
  • Hedging: a cancelled hedge loser no longer records a route-health observation, so it can't count as a latency success or a throttle for learned cooldowns. An ambiguous outcome set earlier on an attempt now stays ambiguous.
  • Canary and stream cost together: the canary scan runs first. If either stops the stream, the client gets one protocol error, the upstream is cancelled once and the unresolved hold stays.
  • routes/unified.py is at 992 lines.

Worker

  • Native direct generations run, in order, latency admission, the stream cost preflight and canary preparation, before cache, reservation and admission.
    • These policies apply only to requests the Worker authenticated itself, never to requests forwarded to Flask.
    • A latency rejection returns 503 before dispatch.
    • A capped streaming key whose model has no known price gets 503 stream_cost_unpriced before dispatch.
    • Canary requests skip the exact and semantic caches.
    • With any of these policies active, a stopped or ambiguous stream is recorded as canceled with unknown cost and no replay permission.
  • Accounts: every native activeUsersByPrefix caller passes env, so the cap column is read only while the stream cost breaker is on.
  • Private domains: learned-cooldown in the control-state dispatcher (get/put, 4 KiB, 404 while its mode is off).
  • Realtime: /v1/realtime* is handled before every new policy, and its usage still reaches native usage metrics.
  • Receipts: the public receipt routes keep forwarding to Flask, which authenticates the models scope.

Known limits

  • Latency coverage: session-lane preparation, precision lookup, health probes and traffic cohorts have unknown coverage at the early boundary. A policy entry with require_coverage returns 503 for those requests instead of guessing; without it they pass through.
  • Messages canary errors: a translated Messages stream keeps the canary error message but may omit the internal error code.
  • Native reroute: a native direct generation names one model, so latency admission there can only pass or reject. Reroute applies on the managed path.
  • Stream cost is not an invoice ceiling: buffered provider work, hidden reasoning and multimedia can push the real cost past the cap.
  • Large files: cloudflare-worker.mjs, routes/proxy.py and services/proxy_service.py received only small call sites.

Before merging to main

Four D1 migrations are included: 0029_realtime.sql through 0032_usage_receipts.sql. A push to main does not apply them. Apply them first:

npx wrangler d1 migrations apply multillm-intelligence --remote

Apply 0030 before setting STREAM_COST_BREAKER_ENABLED: with the flag on and the column missing, account reads return 503.

Tests

  • New tests:
    • tests/test_wave6_registrar.py: hook order, receipt route gating and storage plumbing;
    • tests/test_wave6_managed.py: the managed pipeline through the registered routes;
    • tests/test_wave6_worker.mjs: the native order, private domains, reconciliation receipts and Realtime routing.
  • They cover:
    • all flags off, byte for byte;
    • each flag alone;
    • the combined cases:
      • the canary marker never reaches the idempotency fingerprint, caches, hosted state or logs;
      • a latency rejection happens before any claim or provider call;
      • hedging races only SLO-approved candidates;
      • a hedge loser is not a cooldown throttle;
      • one protocol error and one cancellation when stream cost and canary both fire.
  • Full Python suite: 4,769 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,768 passed. npm run test:integration: 7 passed.
  • cf:migrations:check reports 32 contiguous, additive migrations. test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

Defines frozen identity, tenancy, quota and credit authority contracts with explicit adapters and deny-by-default missing authority, plus an administrator-only dry-run preview at /admin/enterprise/preview. ENTERPRISE_PREVIEW_ENABLED defaults to off, and registering the contracts activates nothing.
Adds the context canary scope hook and latency-SLO admission to the named
post-authentication order, mounts the usage receipt routes behind a pre-auth
404 gate, and connects the learned cooldown and usage receipt stores to the
private D1 endpoints and the reservation reconciliation.

Managed dispatch gets read-only latency candidates, dispatch-only canary
annotation with per-attempt scanning, cache bypass for opted-in requests and
a single upstream cancellation when a policy stops a stream. Native Worker
generations run the latency, stream cost and canary policies for
Worker-authenticated requests only.

Every path is unchanged while its flag is off.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 321fc2d2-0163-4ae4-9b62-017aabae3c5d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
🔵 In progress
View logs
multillm-proxy 546cd41 Oct 09 2026, 10:40 AM

SAML_ENABLED (off by default) adds broker-backed dashboard login. The gateway
never parses SAML XML: an operator-run broker returns a compact JWS that is
verified against pinned RS256 or Ed25519 keys and bound to the issuer,
audience, recipient, nonce and browser state. Each request nonce is claimed
once, and only an operator-created subject link maps a verified subject to an
existing account, with no just-in-time accounts or administrator grants.

Native ACS and metadata return 503 saml_native_unavailable. Request state,
subject links and audit rows live in D1 (migration 0035).
…webhooks

PAYMENTS_ENABLED (off by default) adds POST /v1/payments/checkout, which
creates a Stripe Checkout Session for a whole-cent USD amount between $0.50
and $1,000 from an allowlisted return origin, with per-principal daily limits
and idempotent replay. POST /v1/payments/webhook verifies the raw body's v1
signature and timestamp, claims each event before any callback, and checks
livemode, merchant, currency, amount and session against the stored checkout.

Credits are added only through the injected payment callback, once per
checkout; refunds and disputes append compensation. The browser return page
never credits. Configuration holds secret references only (migration 0038).
…tion

TENANT_GOVERNANCE_ENABLED (off by default) applies organisation and team
policies to requests in a verified workspace. Model and tool grants
intersect with the key's own grants and never widen them. Budgets at every
configured level, plus the key's own limits, are reserved together in one
atomic operation and settled together; an over-limit level denies the
request before dispatch, and an unknown cost keeps its hold at every level.

Roles come from a static permission table. Administrators edit policies
through /admin/organisations/<org_id>[/teams/<team_id>]/governance with
If-Match and audit rows, and /v1/usage gains a workspace breakdown for
callers in a workspace. Requests outside a workspace are unchanged
(migration 0034).
ORGANISATIONS_ENABLED (off by default) adds a two-level organisation and team
hierarchy with stable opaque IDs, roles, caps and deactivation instead of
deletion. Administrators manage it under /admin/organisations with
If-Match revisions and audit rows. Each account has at most one server-side
workspace binding; /v1/workspaces lists the caller's memberships and
/v1/workspaces/switch changes the binding only after a membership check.

A new first post-authentication hook resolves the verified TenantContext and
refuses a missing or deactivated workspace with 404 or 403 before any
storage, cache, claim or provider call. Caller headers never choose a
workspace. Native Worker generations, knowledge and Realtime resolve the
same context, only after the key matches. Accounts without a binding stay
in the legacy namespace (migration 0033).
@ALikesToCode
ALikesToCode merged commit 3fee165 into integrate/gateway-waves-1-5 Oct 9, 2026
6 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the integrate/gateway-waves-1-6 branch October 26, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant